Distributed power management for non-volatile memory controllers using average and peak power credits allocated to memory channels

ABSTRACT

In a memory system having a storage controller and a plurality of distinct sets of non-volatile memory devices, each respective channel controller of a plurality of channel controllers, each channel controller corresponding to a distinct set of the plurality of distinct sets of non-volatile memory devices, receives power credits allocated by the storage controller, including an average power credit and a peak power credit; and while executing commands in the one or more command queues, limits execution of said commands in accordance with the received average power credit and the received peak power credit. In some embodiments, a total number of average power credits allocated by the storage controller is variable and a total number of peak power credits allocated by the storage controller is fixed.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNos. 62/508,319, filed on May 18, 2017, and 62/508,313, filed on May 18,2017 both of which are hereby incorporated by reference in theirentireties.

TECHNICAL FIELD

The disclosed embodiments relate generally to storage controllersystems, and in particular, to credit-based power management in storagesystems.

BACKGROUND

Semiconductor storage systems are commonly used for storing and managingdata for electronic devices. A typical non-volatile data storage systemstores data as an electrical value in the memory cells of the storagesystem and memory controllers are generally tasked with managing datatransactions across multiple memory devices of the storage system.

Data transactions in data storage systems are generally carried out byexecutions of memory commands. To facilitate this process, memorycontrollers are often constructed with command queues that help optimizecommand executions across multiple memory cells. Multiple commandsexecuted in parallel across the storage system, however, can result inspikes in power consumption.

SUMMARY

Various embodiments of systems, methods, and devices within the scope ofthe appended claims each have several aspects, no single one of which issolely responsible for the attributes described herein. Without limitingthe scope of the appended claims, after considering this disclosure, andparticularly after considering the section entitled “DetailedDescription” one will understand how the aspects of various embodimentsare used to manage power consumption in storage devices (e.g.,solid-state drives, sometimes called SSD's). In one aspect, a storagedevice includes a storage controller, and one or more non-volatilememory controllers coupled to the storage controller. The storage devicealso includes a plurality of non-volatile memory devices, eachnon-volatile memory device in the plurality of non-volatile memorydevices coupled to a particular channel controller of the one or morechannel controllers. The storage controller is configured to obtainbacklog information from the channel controllers. The storage device isfurther configured to, in accordance with the obtained backloginformation, allocate credits to channel controllers, and the channelcontrollers limit execution of pending memory commands in accordancewith the allocated power credits.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood in greater detail, amore particular description may be had by reference to the features ofvarious embodiments, some of which are illustrated in the appendeddrawings. The appended drawings, however, merely illustrate pertinentfeatures of the present disclosure and are therefore not to beconsidered limiting, for the description may admit to other effectivefeatures.

FIG. 1 is a block diagram illustrating an implementation of a datastorage system, in accordance with some embodiments.

FIG. 2A is a block diagram illustrating an implementation of a memorychannel, including a channel controller, in accordance with someembodiments.

FIG. 2B is a block diagram illustrating an implementation of amanagement module, in accordance with some embodiments.

FIG. 3A is a block diagram illustrating an implementation of commandqueues and a data structure storing information corresponding to powercredits usage for each command type of a plurality of commands types, inaccordance with some embodiments.

FIG. 3B is a block diagram illustrating acquisition of board powermeasurement(s) and backlog information, which is processed by amanagement module, in accordance with some embodiments.

FIGS. 4A-4B illustrate a flowchart representation of a method ofallocating power credits to one or more channel controllers and limitingexecution of commands in a command queue, in accordance with someembodiments.

FIGS. 5A-5B illustrate a flowchart representation of a method ofallocating power credits to one or more channel controllers and limitingexecution of commands in a command queue, in accordance with someembodiments.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

DETAILED DESCRIPTION

The various embodiments described herein include systems, methods,and/or devices used to manage power consumption in memory systems. Someembodiments include systems, methods, and/or devices to defer executionof pending memory commands.

(A1) In one aspect, a method of operation in a memory system having astorage controller and a plurality of distinct sets of non-volatilememory devices includes, at each respective channel controller of aplurality of channel controllers, each channel controller correspondingto a distinct set of the plurality of distinct sets of non-volatilememory devices and having one or more command queues for holding thechannel controller's pending commands, determining a backlog of therespective channel controller. The method further includes, at eachrespective channel controller, receiving power credits allocated by thestorage controller based at least in part on the backlog of therespective channel controller, and executing commands in the one or morecommand queues of the respective channel controller, including limitingexecution of said commands in accordance with the received powercredits.

(A2) In some embodiments of the method of A1, limiting execution of saidcommands includes deferring execution of a respective command in the oneor more command queues in accordance with a determination that executingthe respective command would require power credits in excess of powercredits available in the respective channel controller.

(A3) In some embodiments of the method of A2, said commands includecommands having different command types, each command type having anassigned number of power credits, and each respective channel controllerdefers execution of a respective command of the respective channelcontroller's pending commands in accordance with a determination thatthe number of power credits assigned to the command type of therespective command exceeds the power credits available in the respectivechannel controller.

(A4) In some embodiments of the method of any of A2-A3, the methodfurther includes, at each respective channel controller, updating thepower credits available in the respective channel controller by reducingthe available power credits when execution of a respective command isinitiated, and increasing the available power credits when execution ofthe respective command completes.

(A5) In some embodiments of the method of any of A2-A3, the methodfurther includes, at each respective channel controller, determining anumber of in use power credits, based on commands currently beingexecuted by the respective channel controller, and determining the powercredits available in the respective channel controller in accordancewith the received power credits allocated by the storage controller andthe in use power credits.

(A6) In some embodiments of the method of any of A1-A5, the methodfurther includes, at the storage controller, receiving backloginformation from each respective channel controller of the plurality ofchannel controllers, and distributing power credits to each respectivechannel controller of the plurality of channel controllers based on atotal number of power credits and the backlog of the respective channelcontroller.

(A7) In some embodiments of the method of A6, the method furtherincludes, at the storage controller, adjusting the total number ofavailable power credits based at least in part on one or more boardpower measurements and/or one or more temperature measurements.

(A8) In some embodiments of the method of any of A1-A7, the methodincludes, at each respective channel controller, determining a backlogscore in accordance with a count of commands whose execution wasdeferred, in an epoch prior to a current epoch, in accordance with adetermination that executing those commands would have required powercredits in excess of power credits available in the respective channelcontroller during the prior epoch.

(A9) In some embodiments of the method of any of A1-A7, the methodincludes, at each respective channel controller, determining a backlogscore based at least in part on pending commands in the one or morecommand queues of the respective channel controller.

(A10) In some embodiments of the method of A9, the method includes, ateach respective channel controller, determining a backlog score based atleast in part on respective ages of one or more of the commands in theone or more command queues of the respective channel controller.

(A11) In some embodiments of the method of any of A1-A10, the one ormore command queues include a high priority queue for read commands andat least one low priority queue for write and erase commands.

(A12) In some embodiments of the method of any of A1-A11, at eachrespective channel controller, receiving during each epoch of a sequenceof epochs, a power credit allocation for the epoch, and limitingexecution of said commands in the one or more command queues, duringeach said epoch, in accordance with the received power credit allocationfor the epoch.

(A13) In some embodiments of the method of any of A1-A12, at eachrespective channel controller, a next command is selected for execution,from among said commands in the one or more command queues, inaccordance with predefined command selection criteria.

(A14) In another aspect, some embodiments include a memory system havinga plurality of distinct sets of non-volatile memory devices, a storagecontroller, and a plurality of channel controllers, each channelcontroller corresponding to a distinct set of the plurality of distinctsets of non-volatile memory devices. Each respective channel controllercontains one or more command queues for holding the respective channelcontroller's pending commands and is configured to determine a backlogof the respective channel controller (e.g., in accordance with pendingcommands in the one or more command queues waiting for execution);receive power credits allocated by the storage controller, based atleast in part on the backlog of the respective channel controller; andexecute commands in the one or more command queues, including limitingexecution of said commands in accordance with the received powercredits.

(A15) In some embodiments, the memory system of A13 is configured toperform the method of any of A2-A13.

(B1) In another aspect, some embodiments include a method of operationin a memory system having a storage controller and a plurality ofdistinct sets of non-volatile memory devices. In some embodiments, themethod includes, at each respective channel controller of a plurality ofchannel controllers, each channel controller corresponding to a distinctset of the plurality of distinct sets of non-volatile memory devices,each respective channel controller having one or more command queues forholding the respective channel controller's pending commands: receivingpower credits allocated by the storage controller, including an averagepower credit and a peak power credit; and executing commands in one ormore command queues, including limiting execution of said commands inaccordance with the received average power credits and the received peakpower credits.

(B2) In some embodiments of the method of B1, the method furthercomprises, at the storage controller, allocating a variable total numberof average power credits and allocating a fixed total number of peakpower credits.

(B3) In some embodiments of the method of B2, the method furtherincludes determining the total number of peak power credits based oncharacteristics of the memory system.

(B4) In some embodiments of the method of any of B2-B3, the methodfurther includes adjusting the total number of average power creditsbased at least in part on one or more board power measurements and/orone or more temperature measurements.

(B5) In some embodiments of the method of B4, the method furthercomprises adjusting the total number of average power credits at fixedtime intervals.

(B6) In some embodiments of the method of B1-B5, limiting executionincludes deferring execution of a respective command in the one or morecommand queues in accordance with a determination that executing therespective command would require average power credits in excess ofaverage power credits available in the respective channel controller orthat executing the respective command would require peak power creditsin excess of peak power credits available in the respective channelcontroller.

(B7) In some embodiments of the method of B6, said commands includecommands having different command types, each command type having anassigned number of average power credits and peak power credits, andeach respective channel controller is configured to defer execution of arespective command of the pending commands in accordance with adetermination that the number of average power credits assigned to thecommand type of the respective command exceeds the average power creditsavailable in the respective channel controller or the peak power creditsassigned to the command type of the respective command exceeds the peakpower credits available in the respective channel controller.

(B8) In some embodiments of the method of any of B6-B7, the methodfurther comprises, at each respective channel controller, updating theaverage power credits and the peak power credits available in therespective channel controller by reducing the available average powercredits and the available peak power credits when execution of arespective command is initiated, and increasing the available averagepower credits and available peak power credits when execution of therespective command completes.

(B9) In some embodiments of the method of any of B6-B7, the methodfurther comprises, at each respective channel controller: (1)determining a number of in use average power credits, based on commandscurrently being executed by the respective channel controller, anddetermining the average power credits available in the respectivechannel controller in accordance with the received average power creditsallocated by the storage controller and the number of in use averagepower credits; and (2) determining a number of in use peak powercredits, based on commands currently being executed by the respectivechannel controller, and determining the peak power credits available inthe respective channel controller in accordance with the received peakpower credits allocated by the storage controller and the number of inuse peak power credits.

(B10) In some embodiments of the method of any of B1-B9, the methodincludes, at each respective channel controller, receiving during eachepoch of a sequence of epochs, an average power credit allocation and apeak power credit allocation for the epoch, and limiting execution ofsaid commands in the one or more command queues, during each said epoch,in accordance with the received average power credit allocation and thereceived peak power credit allocation for the epoch.

(B11) In some embodiments of the method of any of B1-B10, the methodincludes, including, at each respective channel controller of theplurality of channel controllers, determining a backlog of therespective channel controller and providing the determined backlog tothe storage controller; and the average power credit received by eachrespective channel controller is based at least in part on thedetermined backlog provided by the respective channel controller to thestorage controller.

(B12) In some embodiments of the method of B11, each respective channelcontroller determines determine the backlog score of the respectivechannel controller in accordance with a count of commands whoseexecution was deferred by the respective channel controller, in an epochprior to a current epoch, in accordance with a determination thatexecuting those commands would have required power credits in excess ofpower credits available in the respective channel controller during theprior epoch.

(B13) In some embodiments of the method of B11, each respective channelcontroller determines the backlog score of the respective channelcontroller in accordance with pending commands in the one or morecommand queues waiting for execution.

(B14) In another aspect, some embodiments include a memory system havinga plurality of distinct sets of non-volatile memory devices; a storagecontroller; and a plurality of channel controllers, each channelcontroller corresponding to a distinct set of the plurality of distinctsets of non-volatile memory devices. Each respective channel controllercontains one or more command queues and is configured to: receive powercredits allocated by the storage controller, including an average powercredit and a peak power credit; and execute commands in the one or morecommand queues, including limiting execution of said commands inaccordance with the received average power credit and the received peakpower credit.

(B15) In some embodiments of the memory system of B14, the memory systemis further configured to operate in accordance with the method any ofB2-B13.

(B16) In yet another aspect, some embodiments include a non-transitorycomputer-readable storage medium storing one or more programs forexecution by one or more processors of a storage device, the one or moreprograms including instructions for performing any of the methodsdescribed herein.

Numerous details are described herein in order to provide a thoroughunderstanding of the example embodiments illustrated in the accompanyingdrawings. However, some embodiments may be practiced without many of thespecific details, and the scope of the claims is only limited by thosefeatures and aspects specifically recited in the claims. Furthermore,well-known methods, components, and circuits have not been described inexhaustive detail so as not to unnecessarily obscure pertinent aspectsof the embodiments described herein.

FIG. 1 is a block diagram illustrating an implementation of a datastorage system 100, in accordance with some embodiments. While someexample features are illustrated, various other features have not beenillustrated for the sake of brevity and so as not to obscure morepertinent aspects of the example implementations disclosed herein. Tothat end, as a non-limiting example, data storage system 100 includesstorage device 120, which includes storage controller 128, power usagemonitor 124 (e.g., a current sensor or power sensor, and/or one or moretemperature sensors), power supply 126, and one or more memory channels160 (e.g., memory channels 160-1 to 160-m). The storage device 120 isused in conjunction with or includes computer system 110 (e.g., a hostsystem or a host computer).

Computer system 110 is coupled to storage device 120 through dataconnections 101. However, in some implementations computer system 110includes storage device 120 as a component and/or sub-system. Computersystem 110 may be any suitable computer device, such as a personalcomputer, a workstation, a computer server, or any other computingdevice. Computer system 110 is sometimes called a host or host system.In some implementations, computer system 110 includes one or moreprocessors, one or more types of memory, optionally includes a displayand/or other user interface components such as a keyboard, a touchscreen display, a mouse, a track-pad, a digital camera and/or any numberof supplemental devices to add functionality. Further, in someimplementations, computer system 110 sends one or more host commands(e.g., read commands and/or write commands) on control line 111 tostorage device 120. In some implementations, computer system 110 is aserver system, such as a server system in a data center, and does nothave a display and other user interface components.

Storage controller 128 includes host interface 122, management module121, error control module 132, and storage medium interface 138. Storagecontroller 128 is connected to computer system 110 through hostinterface 122 and data connections 101.

Host interface 122 provides storage controller 128 with an interface tocomputer system 110 through data connections 101. Similarly, storagemedium interface 138 provides storage controller 128 with an interfaceto storage medium 161 though connections 103. Connections 103 aresometimes called data connections, but typically convey commands inaddition to data, and optionally convey metadata, error correctioninformation and/or other information in addition to data values to bestored in memory channels 160 and data values read from memory channels160. In some implementations, storage medium interface 138 includes readand write circuitry, including circuitry capable of providing readsignals to memory channels 160 (e.g., signals representing thresholdvoltages to be used when reading data from NAND-type flash memory). Insome embodiments, connections 101 and connections 103 are implemented ascommunication media over which commands and data are communicated, andeach of these connections uses a protocol such as DDR3, SCSI, SATA, SAS,or the like for handling such communications.

In some implementations, management module 121 includes one or moreprocessing units 127 (sometimes herein called CPUs, processors, orhardware processors, and sometimes implemented using microprocessors,microcontrollers, or the like) configured to execute instructions in oneor more programs (e.g., in management module 121). In someimplementations, the one or more processing units 127 are shared by oneor more components within, and in some cases, outside storage controller128. Management module 121 is coupled to host interface 122, errorcontrol module 132, and storage medium interface 138 in order tocoordinate the operation of these components.

Error control module 132 is coupled to host interface 122, managementmodule 121, and storage medium interface 138. Error control module 132is provided to limit the number of uncorrectable errors inadvertentlyintroduced into data. In some embodiments, error control module 132includes an encoder 133 and a decoder 134. Encoder 133 encodes data byapplying an error control code to produce a codeword, which issubsequently stored in non-volatile memory (e.g., in one of NVM devices140, 142). In some embodiments, when the encoded data (e.g., one or morecodewords) is read from non-volatile memory (e.g., in one of NVM devices140, 142, in one or more memory channels 160), decoder 134 applies adecoding process to the encoded data to recover the data, and to correcterrors in the recovered data within the error correcting capability ofthe error control code. For the sake of brevity, an exhaustivedescription of the various types of encoding and decoding algorithmsgenerally available and known to those skilled in the art is notprovided herein.

In some embodiments, error control module 132 includes one or morehardware processing units. In some embodiments, error control module 132is implemented using a hardware state machine, and in some embodiments,error control module 132 is implemented in an application-specificintegrated circuit (ASIC). In some embodiments, error control module 132uses one or more error detection and/or correction schemes, such asHamming, Reed-Solomon (RS), Bose Chaudhuri Hocquenghem (BCH), andlow-density parity-check (LDPC), or the like.

In some embodiments, each memory channel 160 coupled to storagecontroller 128 through connections 103 includes a channel controller130, or alternatively one or more channel controllers, and one or moreNVM devices 140, 142 (e.g., flash memory die). In some embodiments, eachchannel controller 130 includes one or more processing units 202(sometimes herein called CPUs, processors, or hardware processors, andsometimes implemented using microprocessors, microcontrollers, or thelike) configured to execute instructions in one or more programs (e.g.,one or more programs stored in controller memory of the channelcontroller). In some embodiments, NVM devices 140 (e.g., NVM devices140-1 through 140-n), 142 (e.g., NVM devices 142-1 through 142-k) arecoupled to channel controllers 130 through connections that conveycommands in addition to data, and optionally convey metadata, errorcorrection information and/or other information in addition to datavalues to be stored in NVM devices 140, 142 and data values read fromNVM devices 140, 142.

In some embodiments, storage device 120, memory channels 160, and/or NVMdevices 140, 142 are configured for enterprise storage suitable forapplications such as cloud computing, or for caching data stored (or tobe stored) in secondary storage, such as hard disk drives. Additionallyand/or alternatively, storage device 120, memory channels 160, and/orNVM devices 140, 142 are configured for relatively smaller-scaleapplications such as personal flash drives or hard-disk replacements forpersonal, laptop and tablet computers. While in some embodiments NVMdevices 140, 142 are flash memory devices and channel controllers 130are flash memory controllers or solid state storage controllers, inother embodiments storage device 120 may include other types ofnon-volatile memory devices and corresponding controllers.

In some implementations, a respective memory channel 160 of the memorychannels 160-1 to 160-M includes a single NVM device, while in otherimplementations the respective memory channel includes a plurality ofNVM devices. In some implementations, NVM devices 140, 142 includeNAND-type flash memory or NOR-type flash memory. Further, in someimplementations, each channel controller 130 comprises a solid-statedrive (SSD) controller.

In some embodiments, NVM devices 140, 142 are flash memory chips or die,sometimes herein called flash memory devices. Each NVM device includes anumber of addressable and individually selectable blocks. In someimplementations, the individually selectable blocks (sometimes callederase blocks) are the minimum size erasable units in a flash memorydevice. In other words, each block contains the minimum number of memorycells that can be erased simultaneously. Each block is usually furtherdivided into a plurality of pages and/or word lines, for example, 64pages, 128 pages, 256 pages or another suitable number of pages. Eachpage or word line is typically an instance of the smallest individuallyaccessible (readable) portion in a block. In some implementations (e.g.,using some types of flash memory), the smallest individually accessibleunit of a data set, however, is a sector, which is a subunit of a page.That is, a block includes a plurality of pages, each page contains aplurality of sectors, and each sector is the minimum unit of data forreading data from the flash memory device.

In some embodiments, the blocks in each NVM device are grouped into aplurality of zones or planes. Each zone or plane can be independentlymanaged to some extent, which increases the degree of parallelism forparallel operations, such as reading and writing data to NVM devices140, 142.

As noted above, in some embodiments, data is written to a storage mediumin pages, but the storage medium is erased in blocks. As a result, someof the pages in the storage medium may contain invalid (e.g., stale)data, but those pages cannot be overwritten until the entire blockcontaining those pages is erased. In order to write to the pages withinvalid data, the pages (if any) with valid data in that block are readand re-written to a new block and the old block is erased (or put on aqueue for erasing). This process is called garbage collection. Aftergarbage collection, the new block contains the pages with valid data andmay have free pages that are available for new data to be written, andthe old block can be erased so as to be available for new data to bewritten.

A phenomenon related to garbage collection is write amplification. Writeamplification is a phenomenon where the actual amount of physical datawritten to a storage medium (e.g., NVM devices 140, 142 in storagedevice 120) is a multiple of the logical amount of data written by ahost (e.g., computer system 110, sometimes called a host) to the storagemedium. As discussed above, when a block of storage medium must beerased before it can be re-written, the garbage collection process toperform these operations results in re-writing data one or more times.This multiplying effect increases the number of writes required over thelife of a storage medium, which shortens the time it can reliablyoperate. The formula to calculate the write amplification of a storagesystem is given by equation:

$\frac{{amount}\mspace{14mu}{of}\mspace{14mu}{data}\mspace{14mu}{written}\mspace{14mu}{to}\mspace{14mu} a\mspace{14mu}{storage}\mspace{14mu}{medium}}{{amount}\mspace{14mu}{of}\mspace{14mu}{data}\mspace{14mu}{written}\mspace{14mu}{by}\mspace{14mu} a\mspace{14mu}{host}}$

One of the goals of any flash memory based data storage systemarchitecture is to reduce write amplification as much as possible sothat available endurance is used to meet storage medium reliability andwarranty specifications. Higher system endurance also results in lowercost as the storage system may need less over-provisioning. By reducingwrite amplification, the endurance of the storage medium is increasedand the overall cost of the storage system is decreased. Generally,garbage collection is performed on erase blocks with the fewest numberof valid pages for best performance and best write amplification.

During a write operation, host interface 122 receives a write command,which includes data to be stored in storage device 120 from computersystem 110. The received data, sometimes called write data, is encodedusing encoder 133 of storage controller 128 to produce encoded data,typically in the form of one or more codewords. The resulting encodeddata is stored in non-volatile memory of a particular memory channel160.

During a read operation, host interface 122 receives a read command fromcomputer system 110. In response, data read from non-volatile memory ofa particular memory channel 160 is decoded using decoder 134 of storagecontroller 128 to produce decoded data. The resulting decoded data,sometimes called read data, is provided to computer system 110 inresponse to the read command, via host interface 122.

As explained above, a storage medium (e.g., NVM devices 140, 142) isdivided into a number of addressable and individually selectable blocksand each block is optionally (but typically) further divided into aplurality of pages and/or word lines and/or sectors (which aresub-portions of pages). While erasure of a storage medium is performedon a block basis, in many embodiments, reading and programming of thestorage medium is performed on units of memory that are smaller than ablock, such as a page or word line or sector of a page, each of whichhas multiple memory cells (e.g., single-level cells or multi-levelcells). For example, in some embodiments, programming is performed on anentire page. In some embodiments, a multi-level cell (MLC) NAND flashtypically has four possible states per cell, yielding two bits ofinformation per cell. Further, in some embodiments, a MLC NAND has twopage types: (1) lower pages (sometimes called fast pages), and (2) upperpages (sometimes called slow pages). In some embodiments, a triple-levelcell (TLC) NAND flash has eight possible states per cell, yielding threebits of information per cell. Although the description herein uses TLC,MLC, and SLC as examples, those skilled in the art will appreciate thatthe embodiments described herein may be extended to memory cells thathave more than eight possible states per cell, yielding more than threebits of information per cell. In some embodiments, the encoding formatof the storage media (e.g., TLC, MLC, or SLC and/or a chosen dataredundancy mechanism) is a choice made (or implemented) when data isactually written to the storage media.

Flash memory devices (e.g., NVM 140, 142) utilize memory cells (e.g.,SLC, MLC, and/or TLC) to store data as electrical values, such aselectrical charges or voltages. Each flash memory cell typicallyincludes a single transistor with a floating gate that is used to storea charge, which modifies the threshold voltage of the transistor (e.g.,the voltage needed to turn the transistor on). The magnitude of thecharge, and the corresponding threshold voltage the charge creates, isused to represent one or more data values. In some embodiments, during aread operation, a reading threshold voltage is applied to the controlgate of the transistor and the resulting sensed current or voltage ismapped to a data value.

Storage controller 128 is coupled to computer system 110 and channelcontrollers 130. In some embodiments, during a write operation, storagecontroller 128 receives data from computer system 110 through hostinterface 122 and during a read operation, storage controller 128 sendsdata to computer system 110 through host interface 122. Further, hostinterface 122 provides additional data, signals, voltages, and/or otherinformation needed for communication between storage controller 128 andcomputer system 110. In some embodiments, storage controller 128 andhost interface 122 use a defined interface standard for communicationwith computer system 110, such as double data rate type threesynchronous dynamic random access memory (DDR3). In some embodiments,storage device 120 is or includes a solid-state drive implemented as adual in-line memory module (DIMM) device, compatible with a DIMM memoryslot. For example, in some embodiments, storage device 120 is compatiblewith a 240-pin DIMM memory slot using a DDR3 interface specification.

In some embodiments, storage controller 128 and storage medium interface138 use a defined interface standard for communication with memorychannels 160 and their storage controllers 130, such as serial advancetechnology attachment (SATA). In some other embodiments, the deviceinterface used by storage controller 128 and storage medium interface138 to communicate with channel controllers 130 is SAS (serial attachedSCSI), or other storage interface.

In some embodiments, power usage monitor 124 is coupled to and providesboard power measurement(s) and/or temperature measurements to storagecontroller 128. In some embodiments, power usage monitor 124 includessensors and/or circuitry for measuring and monitoring power consumptionby storage device 120 or one or more subsystems of storage device 120,and/or sensors and/or circuitry for measuring temperature of storagedevice 120 or one or more subsystems of storage device 120. In someembodiments, the subsystem for which power and/or temperature ismonitored includes all the memory channels of storage device 120 (e.g.,all the channel controllers 130 and all the NVM devices 140, 142controlled by the channel controllers). In another example, thesubsystem for which power and/or temperature is monitored by power usagemonitor 124 includes all the NVM devices 140, 142 in the memory channelsof storage device 120, but not the channel controllers 130, storagecontroller 128 and host interface 122. Various embodiments of powerusage monitor 124 are described below with reference to FIG. 3B.

In some embodiments, storage device 120 includes power supply 126. Powersupply 126 outputs one or more power supply voltages to storage device120 for use by storage controller 128 and memory channels 160. Powersupply 126 is discussed in more detail below with reference to FIG. 3A.

Optionally, storage device 120 includes various additional features thathave not been illustrated for the sake of brevity and so as not toobscure more pertinent features of the example embodiments disclosedherein, and a different arrangement of features may be possible.Similarly, storage controller 128 may include various additionalfeatures that have not been illustrated for the sake of brevity and soas not to obscure more pertinent features of the example implementationsdisclosed herein, and that a different arrangement of features may bepossible.

FIG. 2A is a block diagram illustrating an implementation of arespective memory channel 160-i, sometimes herein called memory channeli, and storage controller 128, in accordance with some embodiments.Memory channel i is a respective memory channel of a plurality of memorychannels 160-1 to 160-m, as shown in FIG. 1. In some embodiments, memorychannel i includes channel controller 130-i, NVM devices 140-1 through140-n, and storage controller interface 230, which couples memorychannel i to storage controller 128. Optionally, in some embodiments,memory channel i includes temperature sensor 224 to measure thetemperature in memory channel i and to provides temperature information(e.g., temperature measurements) to storage controller 128, for examplevia storage controller interface 230.

In some embodiments, channel controller 130-i includes one or moreprocessing units 202 (sometimes herein called CPUs, processors, orhardware processors, and sometimes implemented using microprocessors,microcontrollers, or the like) for executing modules, programs and/orinstructions stored in memory 206 (sometimes called controller memory orchannel controller memory) and thereby performing processing operations,memory 206, and one or more communication buses 208 for interconnectingthese components. Communication buses 208 optionally include circuitry(sometimes called a chipset) that interconnects and controlscommunications between system components.

Memory 206 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM or other random access solid state memory devices, and mayinclude non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 206 optionally includesone or more storage devices remotely located from processing unit(s)202. Memory 206, or alternately the non-volatile memory device(s) withinmemory 206, comprises a non-transitory computer readable storage medium.In some embodiments, memory 206, or the computer readable storage mediumof memory 206 stores the following programs, modules, and datastructures, or a subset thereof:

-   -   interface module 210, which is used for handling communications        with storage controller 128;    -   command queue(s) 212 that store commands corresponding to        various operations (e.g., read, write and erase) which, when        executed, operate on data held within the memory channel's NVM        devices (e.g., NVM devices 140); in some embodiments, a        respective memory channel has a high priority queue 212-1 and        one or more low priority queues 212-2; in some embodiments, one        or more queues also store command ages (as discussed below with        reference to FIG. 3A);    -   backlog determination module 216, which determines a backlog        based on pending commands in the one or more command queues 212        waiting for execution; and    -   command execution module 218, which dispatches commands from one        or more command queues 212 to NVM devices 140 in memory channel        i for execution; power credits usage per command type        information 220 is used to determine whether to defer execution        of commands in one or more command queues (e.g., command queue        212) in accordance with the command type of the pending command        in the one or more command queues (as discussed below with        reference to FIG. 3A).

In some embodiments, the one or more command queues 212 in a respectivechannel controller 130 are used to hold commands waiting for executionby a set of NVM devices 140 or 142 coupled to the respective channelcontroller 130. A respective command queue 212, when not empty, containsone or more commands corresponding to read, write and/or eraseoperations for reading data from, writing data to, or erasing data froma corresponding set of NVM devices (e.g., NVM devices 140). In someembodiments, commands in command queue(s) 212 include host commandsreceived from computer system 110, while in some other embodiments,commands in command queue(s) 212 include memory operation commandsderived from or determined from host commands received from computersystem 110. For example, multiple memory operation commands may derivedfrom (and thus correspond to) a single host command, and it is themultiple memory operation commands that are stored in one or morecommand queues 212 of one or more memory channels 160. The combinationof a set of NVM devices 140 (e.g., NVM devices 140-1 through 140-n), anda corresponding channel controller 130 (e.g., channel controller 130-i)is sometimes referred to as a memory channel (e.g., memory channel i).Storage device 120 can include as many memory channels as there aredistinct sets of NVM devices to which commands can be dispatched inparallel by a set of channel controllers 130.

In some embodiments, command execution module 218 includes power creditsusage per command type information 218-1 and logic for determiningwhether to defer execution of commands in one or more command queue(s)212. In some embodiments, command execution module 218 limits executionof commands in command queue(s) 212 in accordance with power creditsreceived from storage controller 128. In some embodiments, commandexecution module 218 selects a next command for execution in accordancewith predefined selection criteria. The command selection criteria mayinclude the age of the commands, the order of the commands in eachcommand queue 212, giving priority to commands in a high priority queue212-1, and the amount of power credits required by each command in theone or more command queues (e.g., if a new command requires more powercredits than the available power credits, another command requiringfewer power credits may be selected).

In some embodiments, in a respective memory channel 160-i, commandexecution module 218 dispatches some commands from a respective commandqueue 212 to non-volatile memory devices (e.g., NVM devices 140 or 142)in that memory channel, but defers dispatching other commands from therespective command queue 212, or from another command queue 212, to thenon-volatile memory devices in the memory channel, as further describedbelow with reference to FIGS. 4A-4B and 5A-5B.

In some embodiments, storage controller interface 230 and interfacemodule 210 receive, from storage controller 128, memory operationcommands, such as read, write (also called program), and/or erasecommands, as well as power credits. Further, in some embodiments,storage controller interface 230 and interface module 210 send tostorage controller 128 backlog information, as described in more detailbelow.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 206 maystore a subset of the modules and data structures identified above.Furthermore, memory 206 may store additional modules and data structuresnot described above. In some embodiments, the programs, modules, anddata structures stored in memory 206, or the computer readable storagemedium of memory 206, provide instructions, that when executed by theone or more processors 202, implement at least those portions of themethods described below with reference to FIGS. 4A-4B and 5A-5B that areperformed by or within a respective memory channel i.

Although FIG. 2A shows memory channel i, FIG. 2A is intended more as afunctional description of the various features which may be present in amemory channel than as a structural schematic of the embodimentsdescribed herein. In practice, and as recognized by those of ordinaryskill in the art, items shown separately could be combined and someitems could be separated. In addition, in some other embodiments, one ormore of the functions described herein as being performed by processingunit(s) 202 are instead performed by storage controller 128.

FIG. 2B is a block diagram illustrating an implementation of managementmodule 121, in storage controller 128 (see FIG. 1), in accordance withsome embodiments. Management module 121 typically includes: one or moreprocessing units 127 (sometimes herein called CPUs, processors, orhardware processors, and sometimes implemented using microprocessors,microcontrollers, or the like) for executing modules, programs and/orinstructions stored in memory 254 (sometimes herein called controllermemory) and thereby performing processing operations; memory 254; andone or more communication buses 252 for interconnecting thesecomponents. One or more communication buses 252, optionally, includecircuitry (sometimes called a chipset) that interconnects and controlscommunications between system components.

Management module 121 is operatively coupled to host interface 122,power usage monitor 124, and storage medium interface 138 bycommunication buses 252, and to channel controllers 130 (e.g., channelcontroller 130-1 through 130-m) via storage medium interface 138. Memory254 includes high-speed random access memory, such as DRAM, SRAM, DDRRAM or other random access solid state memory devices, and may includenon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. In some embodiments, memory254 includes one or more storage devices remotely located from the oneor more processing units 127. Memory 254, or alternatively thenon-volatile memory device(s) within memory 254, comprises anon-transitory computer readable storage medium. In some embodiments,memory 254, or the non-transitory computer readable storage medium ofmemory 254, stores the following programs, modules, and data structures,or a subset or superset thereof:

-   -   interface module 256 that is used for communicating with other        components, such as computer system 110 via host interface 122        and memory channels 160 via storage medium interface 138;    -   command module 258 for performing various memory operations        (e.g., sending read, write/program, and/or erase commands to        memory channels 160 in accordance with commands received from        computer system 110, or in accordance with internal operations        such as garbage collection);    -   power usage monitor module 260 for processing board power        measurement(s) (e.g., measurement of total current drawn by        storage device 120 or its memory channel(s)) and/or temperature        measurements received from power usage monitor 124 (FIG. 1) or        from temperature sensor(s) 224 in one or more memory channels;        and    -   power credits module 262, that is used to control a total number        of power credits and to allocate power credits to respective        channel controllers 130 based on backlog information received        from the channel controllers 130, as discussed in more detail        below with reference to FIG. 3B.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 254 maystore a subset of the modules and data structures identified above.Furthermore, memory 254 may store additional modules and data structuresnot described above. In some embodiments, the programs, modules, anddata structures stored in memory 254, or the computer readable storagemedium of memory 254, provide instructions for implementing respectiveoperations in the methods described below with reference to FIGS. 4A-4Band 5A-5B.

Although FIG. 2B shows a management module 121, FIG. 2B is intended moreas a functional description of the various features which may be presentin a management module than as a structural schematic of the embodimentsdescribed herein. In practice, and as recognized by those of ordinaryskill in the art, items shown separately could be combined and someitems could be separated.

FIG. 3A is a block diagram illustrating an implementation of commandqueue(s) 212 and a power credits usage per command type data structure220, in accordance with some embodiments. While some example featuresare illustrated, various other features have not been illustrated forthe sake of brevity and so as not to obscure more pertinent aspects ofthe example embodiments disclosed herein. To that end, as a non-limitingexample, command queue(s) 212 includes a high priority command queue212-1 and a low priority command queue 212-2. In some embodiments, highpriority command queue 212-1 stores commands 302 waiting for execution,and the ages 310 of the respective commands (e.g., read commands andages of read commands). In some embodiments, different types of commandsare stored in different queues (e.g., high priority command queue 212-1stores read commands 302, and low priority queue 212-2 stores write anderase commands 306). Command execution module 218 of a respective memorychannel determines whether to defer execution of commands in the commandqueue(s) (e.g., command queues 212) for that respective memory channel160. In some embodiments, more than two command queues 212 are used tostore commands waiting for execution. In one example, each of three ormore types of commands waiting for execution are stored in differentcommand queues 212.

While in some embodiments a single type of power credit is allocated bythe storage controller to the channel controllers, in some otherembodiments, there are at least two types of power credits allocated bythe storage controller: average power credits and peak power credits. Insome embodiments, the total number of average power credits available tobe allocated is adjusted based on power usage and/or temperaturemeasurements, when those measurements indicate that power usage by thestorage device is above a threshold or is in danger of rising above thethreshold, as described in more detail with reference to FIG. 3B.

On the other hand, in some embodiments, the total number of peak powercredits to be allocated by the storage controller is static. The totalnumber of peak power credits is based on a characterization of thestorage device and the ability of the storage device's power supply 126to handle power usage spikes. The purpose of peak power credits is tolimit the maximum size of power usage spikes that can occur to amagnitude that can be handled by the device. When operations areoccurring on many die and hit their peak current simultaneously, thepower supply voltage at the storage device may drop below a predefinedthreshold level, which defines the lowest safe operating voltage for theoperating the storage device (e.g., 0.5V below the nominal supplyvoltage), if the peak current used by the storage device exceeds amaximum load, or a rate of change of the current used by the storagedevice exceeds a maximum rate. In some embodiments, the use of a peakpower credit limit ensures that the power supply voltage at the storagedevice does not drop below the threshold level, below which operation ofthe storage device may be adversely impacted.

In some embodiments, power credits usage per command type data structure220 stores information based on different command types (e.g., readcommands, write commands, erase commands). In some embodiments, theinformation stored for each type of command includes an average powercredit value 314 and a peak power credit value 316. For example, in someembodiments, power credits usage per command type data structure 220 isa table having information (e.g., in a record) for each of a pluralityof commands types (e.g., command types 1 to p). In some embodiments, thetable includes a single power credit usage value 314 (shown in FIG. 3Aas the “average” power credit usage) for each command type, while inother embodiments the table includes two distinct power credit usagevalues (e.g., average power credit usage 314, and peak power creditusage 316) for each command type. The distinction between average powercredit usage and peak power credit usage is explained below withreference to FIGS. 5A-5B. In some embodiments, each type of command isassigned a different number of power credits. Examples of the subsystemfor which power is monitored are discussed below with reference to FIG.3B.

FIG. 3B is a block diagram illustrating an acquisition of board powermeasurement(s) 320 and pending command backlogs processed by amanagement module, in accordance with some embodiments. While someexample features are illustrated, various other features have not beenillustrated for the sake of brevity and so as not to obscure morepertinent aspects of the example embodiments disclosed herein. Inparticular, the mechanisms shown in FIG. 3B will first be described forembodiments in which a single type of power credit is allocated amongthe channel controllers 130, and thereafter will be described forembodiments in which two or more types of power credit are allocatedamong the channel controllers 130.

As a non-limiting example, board power measurements and/or temperaturemeasurements 320 (e.g., measurements of total current drawn by thememory channels and/or temperature measurements), are collected by powerusage monitor 124. The collected measurements are received by powerusage monitor module 260, and used by power credits module 262 todetermine and update the total power credits available, or in someembodiments that use two or more types of power credits, power creditsmodule 262 determines and updates the total average power creditsavailable. It is noted that while a measurement of the total currentdrawn by a storage device or its memory channels is not, strictlyspeaking, the same as a measurement of power usage, it is effectively apower measurement because the voltage level at which the current isprovided to the storage device is a known, substantially fixed, voltagelevel. Stated another way, fluctuations in the voltage level of thesupplied power are sufficiently small during ordinary usage of thestorage device that measurements of current drawn are a good proxy formeasurements of power usage. Therefore, in some embodiments, power ismeasured by measuring total current drawn by the storage device, thestorage device's memory channels, or some other predefined portion ofthe storage device.

In some embodiments, the total power credits available are determined asa function of the received power measurement(s):TotalCredits=TPCfunction(power measurement(s))where TotalCredits are the total power credits available (e.g., inembodiments using a single type of power credits), and TPCFunction( )maps the power and/or temperature measurements 320 to a value of thetotal power credits available. In some embodiments, TPCFunction( ) is anon-linear function, having a default value (e.g., a value configured orset during manufacturing or initial testing of the storage device) forvalues of the power and/or temperature measurement(s) 320 below apredefined threshold for embodiments having a single power/temperaturemeasurement, or below a set of predefined thresholds (e.g., onethreshold for total current drawn by storage device or its memorychannels and another threshold for the temperature), and values thatdecrease from the default value linearly or non-linearly for values ofthe power/temperature measurement(s) above the predefined threshold(s).In one example, in which total current drawn and temperature are bothmonitored, TPCFunction( ) is equal to the lesser of two values, (A)defaultValue−Fcn1(MeasuredCurrent), and (B)defaultValue−Fcn2(MeasuredTemperature), where defaultValue is the amountof power credits available for allocation (e.g., in a respective epoch)when both the measured current and measured temperature are below theirrespective thresholds, Measured Current is the measured current, Fcn1 isa function of the measured current that has a positive non-zero valueonly when measured current is above a predetermined threshold current,MeasuredTemperature is the measured temperature, and Fcn2 is a functionof the measured temperature that has a positive non-zero value only whenmeasured temperature is above a predetermined threshold temperature. Insome embodiments, other functions (e.g., nonlinear functions) are usedto determine the total power credits available for allocation.

When the power/temperature measurement(s) exceed the predefinedthreshold(s), this indicates that the storage device is using morepower, on average, than allowed, and therefore the total power creditsavailable is reduced. Reducing the total power credits available willresult in fewer commands being executed per unit of time (sometimescalled epochs), which will reduce the total or average power used by thestorage device, which, in turn, will reduce the temperature of thestorage device or the portion of the storage device monitored by powerusage monitor 124.

It is noted that operating the storage device at a temperature above acritical temperature may result in failure of the storage device toretain data or otherwise operate properly. Therefore, to prevent thatfrom happening, in embodiments that monitor temperature of the storagedevice, or monitor temperature of one or more portions or components ofthe storage device (e.g., using temperature sensors 224 in memorychannels 160, see FIG. 2A), a threshold temperature that is below thecritical temperature associated with device failure is used as thepredetermined threshold temperature. In some embodiments that measuretemperature at more than one location within the storage device, e.g.,by measuring temperature of two or more portions (e.g., memory channels160) or components of the storage device, the highest measuredtemperature of the two or more portions or components of the storagedevice is used as the measured temperature; in some other embodiments,an average or other predefined combination of the measured temperaturesof the two or more portions or components of the storage device is usedas the measured temperature. When the measured temperature is above thepredetermined threshold temperature, total power credits allocated bystorage controller 128 (e.g., management module 121, or power creditsmodule 262) are reduced.

As explained in more detail below, in some embodiments, power creditsare determined and allocated by storage controller 128 for successivepredetermined time periods, often called epochs, in a sequence ofpredetermined time periods. For example, the sequence of predeterminedtime periods may be successive time periods of duration T1 (e.g., 1second, or more generally a value in the range of 0.1 second to 10seconds), and for each such time period storage controller 128determines the total power credits available for allocation, and thespecific power credit(s) to be allocated to each storage controller (or,equivalently, to each memory channel).

Above, the determination of total power credits available has beendescribed for embodiments in which a single type of power credit isallocated by the storage controller to the channel controllers (or,equivalently, the memory channels). In some embodiments in which two ormore types of power credits are allocated by the storage controller tothe channel controllers, the above-described TPCFunction( ) maps thepower and/or temperature measurements to a value of the total averagepower credits available for allocation. In some such embodiments, thetotal number of average power credits available to be allocated isadjusted, based on power usage measurements and/or temperaturemeasurements for storage device 120 or a subsystem of storage device 120(e.g., temperature measurements by power usage monitor 124), when thosemeasurements indicate that the temperature of the storage device, or asubsystem of the storage device, is above a threshold temperature or isin danger of rising above the threshold temperature, as described inmore detail with reference to FIG. 3B. In particular, in someembodiments, “temperature throttling” is accomplished by reducing thetotal number of power credits to be allocated (in embodiments using asingle type of power credit) or reducing the number of average powercredits to be allocated (in embodiments using two or more types of powercredits) when the temperature of the storage device is above a thresholdtemperature. In some embodiments, temperature throttling is alsoperformed to reduce the total power credits available, or the number ofaverage power credits available, when the rate of change of the measuredtemperature of the storage device, or a subsystem of the storage device,is above a rate threshold.

In some embodiments, power credits module 262 receives backloginformation from channel controllers 130. At the channel controllers130, backlog determination modules 216 determine, for each memorychannel, a backlog corresponding to the number, type and age of thepending commands in the command queues of that memory channel. Asexplained above with reference to FIG. 3A, in some embodiments, pendingread commands are placed in high priority command queue 212-1, whileother commands (e.g., write and erase commands) are placed in lowpriority command queue 212-2 (or, alternatively, in one or more lowpriority command queues). Further, for purposes of determining thebacklog in each memory channel, some commands, such as high priorityread commands, are assigned different weights based on age (e.g., howlong each such command has been pending, as indicated by a timestampvalue for the command). As described above with reference to in FIG. 3A,in some embodiments, command age information 310 is stored for eachcommand in high priority command queue 212-1. In some embodiments, abacklog score is generated and conveyed by each the channel controller130 of each memory channel to power credits module 262 of storagecontroller 128. For example, in some embodiments, the backlog scoregenerated by the channel controller 130 of a respective memory channelis:Backlog=w*(Σ_(i)age_(i))+LP_queue_levelwhere w is a weighting factor, i is an index for the pending commands inthe high priority command queue 212-1, each age_(i), value is the age ofa corresponding high priority command, as measured in predefined unitsof time (e.g., number of microseconds), and LP_queue_level is the numberof commands in the low priority command queue 212-2. In another example,the backlog score generated by the channel controller 130 of arespective memory channel is:Backlog=(Σ_(i)weight(age_(i)))+LP_queue_levelwhere weight( ) is a function that maps age values to weights, i is anindex for the pending commands in the high priority command queue 212-1,each age_(i) value is the age of a corresponding high priority command,as measured in predefined units of time (e.g., number of microseconds),and LP_queue_level is the number of commands in the low priority commandqueue 212-2. In some embodiments, the weight( ) function is a non-linearfunction, for example a non-linear function that maps ages close to apredefined limit to significantly higher weights than ages that are farbelow the predefined limit. For example, if the predefined limit is 1millisecond, any age between 10 microseconds and 500 microseconds is bemapped to a value of n*age, and any age above 500 microseconds is mappedto a value of 2*n*age, where n is a fixed scaling value.

In some other embodiments, the backlog information conveyed by thechannel controller 130 of each memory channel 160 to power creditsmodule 262 of storage controller 128 includes information about thenumber of pending commands in the corresponding memory channel's lowpriority command queue 212-1 and the number and age of pending commandsin the corresponding memory channel's high priority command queue 212-2.In some such embodiments, storage controller 128 (e.g., power creditsmodule 262 in storage controller 128) generates a backlog score for eachrespective memory channel based on the backlog information received fromthe channel controller 130 of the respective memory channel.

In accordance with the backlog information received from the channelcontrollers (e.g., received by power usage monitor module 260), and thetotal power credits available (e.g., determined by power credits module262), power credits module 262 allocates power credits (e.g., averagepower credits and peak power credits) to respective channel controllers.

In some embodiments, the power credits allocated to a respective memorychannel, or its channel controller, are based on its proportion of theoverall backlog score for all the memory channels in the storage device.For example, in some embodiments, the power credits allocated to anyparticular memory channel is:

${{PowerCreditAllocated}(c)} = {{TotalCredits}*\frac{{BacklogScore}_{m}}{\Sigma_{i}{BacklogScore}_{i}}}$where c identifies the memory channel to which the power credit isallocated, i is an index for the memory channels (e.g., memory channels1 to m), TotalCredits are the total power credits available, andBacklogScore_(i) is the backlog score obtained from channel controlleri.

FIGS. 4A-4B illustrate a flowchart representation a method 400 ofallocating power credits to one or more channel controllers in a memorysystem (e.g., storage device 120) and limiting execution of commands inone or more command queues, in accordance with some embodiments. Method400 is performed in a non-volatile memory system (e.g., storage device120, FIG. 1), which includes a plurality of memory channels 160, eachmemory channel including a channel controller 130 and distinct sets ofnon-volatile memory devices (e.g., NVM devices 140-1 through 140-n, andNVM 142-1 through 142-k), and which coordinates and manages multiplesub-system components to limit execution of commands in the commandqueues of two or more memory channels in accordance with power creditsallocated to those memory channels. More specifically, in someembodiments, some portions of method 400 are performed by the channelcontroller of each of two or more of the memory channels, while otherportions are performed by the system's storage controller.

In some embodiments, a respective channel controller determines abacklog of the respective channel controller (402), as described abovein further detail with respect to FIG. 3B. In some embodiments, asexplained above in more detail with reference to FIG. 3B, a backlogscore is calculated, based on pending commands in the one or morecommand queues of the channel controller, by giving different weights topending commands based on command type and age of the pending commands(404). In some embodiments, the one or more queues of the respectivechannel controller include a high priority queue for read commands and alow priority queue for write and erase commands (406). Optionally,pending commands in the high priority queue (e.g., read commands) aregiven more weight in the backlog calculation than pending commands inthe low priority queue (e.g., write and erase commands).

In some embodiments, the aforementioned backlog determination isperformed by each respective channel controller of the plurality ofchannel controllers in the memory system in accordance with pendingcommands (i.e., command waiting for execution) in the one or morecommand queues of that respective channel controller. In someembodiments, each respective channel controller of the plurality ofchannel controllers in the memory system is configured to determine abacklog score based at least in part on respective ages of one or moreof the commands in the one or more command queues of the respectivechannel controller.

In some embodiments, each respective channel controller of the pluralityof channel controllers in the memory system is configured to determine abacklog score in accordance with a count of commands whose execution wasdeferred by the respective channel controller, in an epoch prior to acurrent epoch (e.g., the epoch immediately prior to the current epoch),in accordance with a determination (e.g., a determination for each suchcommand) that executing those commands would have required power creditsin excess of power credits available in the respective channelcontroller during that prior epoch. Deferral of command execution isdiscussed below with respect to operations 420-426.

Furthermore, in some embodiments, the storage controller receivesbacklog information from the respective channel controller (408). At thestorage controller, as shown in FIG. 3B, power credits module 262receives the backlog information and allocates a portion of the totalnumber of available power credits based on backlog information for therespective channel controller (410). As a non-limiting example, if thebacklog for channel controller 130-1 is lower (or smaller) than thebacklog for channel controller 130-2, more power credits are allocatedto channel controller 130-2 than channel controller 130-1. Furthermore,in some embodiments, distribution of power credits is limited by thetotal number of available power credits, which is adjusted in accordancewith board power measurements described in FIG. 3B (412).

As explained in more detail above with reference to FIG. 3B, in someembodiments the total number of available power credits is adjusted, ordetermined, at the storage controller based at least in part on one ormore temperature measurements. Alternatively, or in addition, the totalnumber of available power credits is adjusted, or determined, at thestorage controller based at least in part on one or more board powermeasurements.

As explained in more detail above, with reference to FIG. 3B, thestorage controller controls (e.g., determines) a total number ofavailable power credits (e.g., based on measured power usage and/ormeasured temperature) and distributes power credits to the respectivechannel controller of the plurality of channel controllers based on thetotal number of power credits and the backlog of the respective channelcontroller. In some embodiments, the distribution of power credits isbased on the proportion of the backlog of each memory channel controllercompared to the sum of the backlogs across all memory channelcontrollers, sometimes called a pro rata distribution of the availablepower credits. Furthermore, the storage controller allocates powercredits to the respective channel controller for each epoch of asequence of epochs, in accordance with some embodiments.Correspondingly, the respective channel controller, during each epoch inthe sequence of epochs, receives a power credit allocation for the epoch(416).

In some embodiments, each respective channel controller limits executionof commands in accordance with the received power credits (420). Theexecution of commands may include the execution of the commands by thenon-volatile memory devices controlled by the channel controller.Furthermore, in some embodiments, each respective channel controller isconfigured to perform operations during each epoch of the sequence ofepochs, including receiving a power credit allocation for the epoch (see416), and limiting execution of commands in the one or more commandqueues of the channel controller, during the epoch, in accordance withthe received power credit allocation for the epoch (422).

In some embodiments, the channel controller defers execution of acommand if executing said command would require power credits in excessof the power credits available in the channel controller (426) (e.g.,power credits available can be the total number of power creditsallocated to the respective channel controller during said epoch, minusthe number of power credits assigned to command(s) currently beingexecuted in the respective channel controller). As described in FIG. 3A,each command type has an assigned number of power credits (428), andexecution of a respective command is deferred if the number of powercredits assigned to the command type of the respective command exceedsthe power credits available.

In some embodiments, each respective channel controller updates thepower credits available in the respective channel controller by reducingthe available power credits when execution of a respective command isinitiated (e.g., by subtracting the number of power credits assigned tothe command(s) currently being executed in the corresponding memorychannel), and increasing the available power credits when execution ofthe respective command completes (e.g., by adding the number of powercredits assigned to the respective command whose execution hascompleted) (432). Alternatively, in some embodiments, each respectivechannel controller updates the power credits available in the respectivechannel controller by setting, at the beginning of each epoch, the powercredits available in the respective channel controller to the powercredits allocated to the respective channel controller for the currentepoch, and subtracting the number of power credits assigned tocommand(s) whose execution is initiated during the current epoch.

From another viewpoint, in some embodiments, each respective channelcontroller determines a number of in use power credits, based oncommands currently being executed by the respective channel controller(e.g., based on the number of power credits assigned to the command(s)currently being executed in the corresponding memory channel), anddetermines the power credits available in the respective channelcontroller in accordance with the received power credits allocated bythe storage controller (e.g., allocated by operations 410, 412, 414,416) and the in use power credits (432).

In some embodiments, the channel controller selects (424) a next commandfor execution in accordance with predefined command selection criteria.For example, in some embodiments, so long as there are sufficient powercredits available, the channel controller selects a next command (fromamong the pending commands in the one or more command queues) forexecution in accordance with a predefined priority scheme. Moreparticularly, in one example, the priority scheme is to execute commandsin the high priority queue first, in order of age (i.e., executing theoldest commands in the high priority queue first), before executingcommands in the low priority queue. However, in some embodiments, if thenext command that would be selected in accordance with the priorityscheme would use more power than the available power credits, thechannel controller selects for execution another pending command, ifany, for which the available power credits are sufficient (i.e., wherethe selected pending command would use no more power credits than theavailable power credits).

FIGS. 5A-5B illustrate a flowchart representation a method 500 ofallocating average power credits and peak power credits to one or morechannel controllers and limiting execution of commands in one or morecommand queues, in accordance with some embodiments. Method 500 isperformed in a non-volatile memory system (e.g., storage device 120,FIG. 1) having a storage controller and a plurality of distinct sets ofnon-volatile memory devices. The non-volatile memory system includes aplurality of memory channels 160, each memory channel including achannel controller 130 and a distinct set of non-volatile memory devices(e.g., NVM devices 140-1 through 140-n, or NVM 142-1 through 142-k) ofthe plurality of distinct sets of non-volatile memory devices. Thenon-volatile memory system coordinates and manages multiple sub-systemcomponents to defer execution of commands in the command queues of twoor more memory channels, each of which corresponds to a distinct set ofnon-volatile memory devices. More specifically, in some embodiments,some portions of method 500 are performed by the channel controller ofeach of two or more of the memory channels, while other portions areperformed by the memory system's storage controller. The memory systemthat performs method 500 also typically includes an interface (e.g.,host interface 122, FIGS. 1 and 2B) for coupling the memory system toone or more host systems, as shown in FIG. 1, and the commands pendingin the one or more command queues of the memory channels includecommands corresponding to memory access commands received from the oneor more most systems.

In some embodiments, a respective channel controller 160 determinesbacklog information for that channel controller's memory channel (502).Methods for determining the backlog information are discussed above withrespect to FIGS. 3A-3B and 4A-4B; the embodiments discussed above withrespect to method 400, with respect to backlog determination, areequally applicable to method 500. In some embodiments, the storagecontroller receives backlog information from the respective channelcontroller (503), determines average power credits and peak powercredits, and allocates the average power credits and peak power creditsto channel controllers 130 (504), or to the corresponding memorychannel. Average power credits and peak power credits are describedabove in more detail with reference to FIG. 3A. In some embodiments,average power credits and peak power credits are each allocated using amethod that is similar in some respects to the method described abovewith reference to FIGS. 4A-4B. In some embodiments, two power creditallocations (e.g., average power credit allocation and peak power creditallocation) are determined for each memory channel as follows:

${{AvgPowerCreditAllocated}(c)} = {{TotalAvgCredits}*\frac{{BacklogScore}_{m}}{\Sigma_{i}{BacklogScore}_{i}}}$${{PeakPowerCreditAllocated}(c)} = {{TotalPeakCredits}*\frac{{BacklogScore}_{m}}{\Sigma_{i}{BacklogScore}_{i}}}$where c identifies the memory channel to which the power credit isallocated, i is an index for the memory channels (e.g., memory channels1 to m), TotalAvgCredits is the total number of average power creditsavailable, TotalPeakCredits is the total number of peak power creditsavailable, and BacklogScore_(i) is the backlog score obtained from (or,alternatively, determined for) channel controller i.

In some embodiments, the total number of available average power creditsis variable, and is adjusted based on one or more board powermeasurement(s) and/or temperature measurement(s) (506), as describedabove in more detail with reference to FIG. 3B. In some embodiments, theadjustment occurs at fixed time intervals (508) (e.g., 10 times persecond, or 1 time per second). In some embodiments, the total number ofavailable peak power credits is fixed, based on characteristics of thememory system (510) (e.g., power spike handling of power supply). Insome embodiments, the allocation of peak power credits is based on thepeak power credit needs of commands waiting for execution (also calledpending commands) in each memory channel. Furthermore, in someembodiments, the storage controller allocates average power credits andpeak power credits to the respective channel controller for each epochof a sequence of epochs. Correspondingly, the respective channelcontroller, during each epoch in the sequence of epochs, receives anaverage power credit allocation and a peak power credit allocation forthe epoch (516). From a system viewpoint, each respective channelcontroller of a plurality of channel controllers in the memory system(e.g., storage device 120) receives power credits allocated by thestorage controller, including an average power credit and a peak powercredit. It is noted that each such channel controller corresponds to adistinct set of the plurality of distinct sets of non-volatile memorydevices in the memory system.

In some embodiments, each respective channel controller limits executionof commands in accordance with the received average power credits andthe received peak power credits (518). The execution of commands mayinclude the execution of the commands by the non-volatile memory devicescontrolled by the channel controller. Furthermore, in some embodiments,each respective channel controller is configured to perform operationsduring each epoch of the sequence of epochs (see 516), includingreceiving an average and peak power credit allocation for the epoch, andlimiting execution of said commands in the one or more command queues inaccordance with the received average power credit allocation and peakpower credit allocation for the epoch (520).

In some embodiments, limiting execution of the pending commands (518)includes deferring execution of a command if executing the command wouldrequire average power credits in excess of the average power creditsavailable in the channel controller (522) or executing the command wouldrequire peak power credits in excess of the peak power credits available(526) in the channel controller. As described above with reference toFIG. 3A, in some embodiments, each command type has an assigned numberof average power credits and peak power credits (524, 528), andexecution of a respective command is deferred if the number of averagepower credits assigned to the command type of the respective commandexceeds the average power credits available (522) or if the number ofpeak power credits assigned to the command type of the respectivecommand exceeds the peak power credits available (526). Thus, if eithertype of power usage limit would be exceeded by execution of therespective command, execution of the respective command is deferred.

In some embodiments, the average power credits available in a respectivechannel controller are the total number of average power creditsallocated to the respective channel controller during the current epoch,minus the number of average power credits assigned to command(s)currently being executed in the respective channel controller.Similarly, in some embodiments, the peak power credits available in arespective channel controller are the total number of peak power creditsallocated to the respective channel controller during the current epoch,minus the number of peak power credits assigned to command(s) currentlybeing executed in the respective channel controller.

In some other embodiments, the average power credits available in arespective channel controller are the total number of average powercredits allocated to the respective channel controller during thecurrent epoch, minus the number of average power credits assigned tocommand(s) whose execution was initiated during the current epoch.Similarly, in some embodiments, the peak power credits available in arespective channel controller are the total number of peak power creditsallocated to the respective channel controller during the current epoch,minus the number of peak power credits assigned to command(s) whoseexecution was initiated during the current epoch. In some embodiments,for each type of command, the average power credits assigned to thecommand type and the peak power credits assigned to the command type arethe same, while in other embodiments these are distinct values.

In addition, in some embodiments, the epochs for peak power credits andthe epochs for average power credits have different durations. Forexample, in some embodiments, the epochs for average power credits havea duration that is L times as long as the epochs for peak power credits,where L is a value between two and ten. As a result, in suchembodiments, operation 410 for allocating power credits is performedmore often for allocating peak power credits to each of the respectivechannel controllers than it is performed by for allocating average powercredits to each of the respective channel controllers.

In some embodiments, each respective channel controller updates theaverage power credits available and the peak power credits available inthe respective channel controller by reducing the available average andpeak power credits when execution of a respective command is initiated(e.g., by subtracting the number of average power credits assigned tothe command(s) currently being executed in the corresponding memorychannel from the average power credits available, and subtracting thenumber of peak power credits assigned to the command(s) currently beingexecuted in the corresponding memory channel from the peak power creditsavailable), and increasing the available average and peak power creditswhen execution of the respective command completes (e.g., by adding thenumber of average power credits assigned to the respective command whoseexecution has completed to the available average power credits in thecorresponding memory channel, and adding the number of peak powercredits assigned to the respective command whose execution has completedto the available peak power credits in the corresponding memory channel)(532).

From another viewpoint, in some embodiments, each respective channelcontroller maintains two power credit pools, an average power creditpool and a peak power credit pool. The respective channel controllermaintains a distinct available power credit level for each of the twopools. Initial values of the average and peak available power creditlevels for the two power credit pools, which are also maximum values forthe two power credit pools, are determined in accordance with theconsiderations discussed above. Each time a command is dequeued from acommand queue for execution, the average and peak available power creditlevels for the two power credit pools are updated by subtracting thecorresponding number of average and peak power credits assigned to thatcommand, respectively. In some embodiments, the same numbers of averageand peak power credits are added back to the two power credit pools whenexecution of the command completes. In some other embodiments, however,average and peak power credits are not added back to the two powercredit pools when execution of the command completes, and instead theaverage power credit pool is periodically restored to a first level(e.g., a level equal to the average power credits allocated by thestorage controller to the respective channel controller or its memorychannel for the current epoch) each time a time period having a firstpredetermined duration expires and similarly the peak power credit poolis periodically restored to a second level (e.g., a level equal to thepeak power credits allocated by the storage controller to the respectivechannel controller or its memory channel for the current epoch) eachtime a time period having a second predetermined duration expires.Typically, the time periods with the first predetermined duration (forthe average power credit pool) are longer than the time periods with thesecond predetermined duration (for the peak power credit pool).

In some embodiments of method 500, the channel controller selects (534)a next command for execution. For example, in some embodiments, so longas there are sufficient average power credits available and peak powercredits available, the channel controller selects a next command (fromamong the pending commands in the one or more command queues) forexecution in accordance with a predefined priority scheme. Moreparticularly, in one example, the priority scheme is to execute commandsin the high priority queue first, in order of age (i.e., executing theoldest commands in the high priority queue first), before executingcommands in the low priority queue. However, in some embodiments, if thenext command that would be selected in accordance with the priorityscheme would use more average power credits than the available averagepower credits, or if the next command that would be selected inaccordance with the priority scheme would use more peak power creditsthan the available peak power credits, the channel controller selectsfor execution another pending command, if any, for which the availableaverage power credits and available peak power credits are sufficient(i.e., where the selected pending command would require no more averagepower credits than the available average power credits and no more peakpower credits than the available peak power credits).

From another viewpoint, in some embodiments, each respective channelcontroller determines a number of in use average power credits and inuse peak power credits, based on commands currently being executed bythe respective channel controller (e.g., based on the number of averagepower credits and peak power credits assigned to the command(s)currently being executed in the corresponding memory channel), anddetermines the average power credits available and the peak powercredits available in the respective channel controller in accordancewith the received average power credits and peak power credits allocatedby the storage controller and the in use average power credits and thein use peak power credits (532).

It will be understood that, although the terms “first,” “second,” etc.may be used herein to describe various elements, these elements shouldnot be limited by these terms. These terms are only used to distinguishone element from another. For example, a first device could be termed asecond device, and, similarly, a second device could be termed a firstdevice, which changing the meaning of the description, so long as alloccurrences of the “first device” are renamed consistently and alloccurrences of the second device are renamed consistently. The firstdevice and the second device are both devices, but they are not the samedevice.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the claims. Asused in the description of the embodiments and the appended claims, thesingular forms “a”, “an” and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willalso be understood that the term “and/or” as used herein refers to andencompasses any and all possible combinations of one or more of theassociated listed items. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in accordance with a determination”or “in response to detecting,” that a stated condition precedent istrue, depending on the context. Similarly, the phrase “if it isdetermined [that a stated condition precedent is true]” or “if [a statedcondition precedent is true]” or “when [a stated condition precedent istrue]” may be construed to mean “upon determining” or “in response todetermining” or “in accordance with a determination” or “upon detecting”or “in response to detecting” that the stated condition precedent istrue, depending on the context.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the claims to the precise forms disclosed. Many modifications andvariations are possible in view of the above teachings. The embodimentswere chosen and described in order to best explain principles ofoperation and practical applications, to thereby enable others skilledin the art.

What is claimed is:
 1. A memory system, comprising: a plurality ofdistinct sets of non-volatile memory devices; a storage controller; aplurality of channel controllers, each respective channel controllercorresponding to a distinct set of the plurality of distinct sets ofnon-volatile memory devices, each respective channel controller havingone or more command queues for holding the respective channelcontroller's pending commands and configured to: receive power creditsallocated by the storage controller, wherein each respective channelcontroller is configured to receive an average power credit and a peakpower credit; execute commands in the one or more command queues,including limiting execution of said commands in accordance with thereceived average power credit and the received peak power credit,wherein each respective channel controller is configured to limitexecution of said commands based on a determination that a number ofreceived average power credits and/or peak power credits meets excesspower credit criteria; and wherein a total number of average powercredits allocated by the storage controller is variable and a totalnumber of peak power credits allocated by the storage controller isfixed.
 2. The memory system of claim 1, wherein the total number of peakpower credits at the storage controller is based on characteristics ofthe memory system.
 3. The memory system of claim 1, wherein the storagecontroller is configured to adjust the total number of average powercredits based at least in part on one or more board power measurements.4. The memory system of claim 1, wherein the storage controller isconfigured to adjust the total number of average power credits based atleast in part on one or more temperature measurements.
 5. The memorysystem of claim 3, wherein the adjustment of the total number of averagepower credits at the storage controller occurs at fixed time intervals.6. The memory system of claim 1, wherein limiting execution includesdeferring execution of a respective command in the one or more commandqueues in accordance with a determination that executing the respectivecommand would require average power credits in excess of average powercredits available in the respective channel controller or that executingthe respective command would require peak power credits in excess ofpeak power credits available in the respective channel controller. 7.The memory system of claim 6, wherein said commands include commandshaving different command types, each command type having an assignednumber of average power credits and peak power credits, and eachrespective channel controller is configured to defer execution of arespective command of the respective channel controller's pendingcommands in accordance with a determination that the number of averagepower credits assigned to the command type of the respective commandexceeds the average power credits available in the respective channelcontroller or the peak power credits assigned to the command type of therespective command exceeds the peak power credits available in therespective channel controller.
 8. The memory system of claim 6, whereineach respective channel controller is configured to update the averagepower credits and the peak power credits available in the respectivechannel controller by reducing the available average power credits andthe available peak power credits when execution of a respective commandis initiated, and increasing the available average power credits andavailable peak power credits when execution of the respective commandcompletes.
 9. The memory system of claim 6, wherein each respectivechannel controller is configured to: determine a number of in useaverage power credits, based on commands currently being executed by therespective channel controller, and determine the average power creditsavailable in the respective channel controller in accordance with thereceived average power credits allocated by the storage controller andthe number of in use average power credits; and determine a number of inuse peak power credits, based on commands currently being executed bythe respective channel controller, and determine the peak power creditsavailable in the respective channel controller in accordance with thereceived peak power credits allocated by the storage controller and thenumber of in use peak power credits.
 10. The memory system of claim 1,wherein each respective channel controller is configured to receive,during each epoch of a sequence of epochs, an average power creditallocation and a peak power credit allocation for the epoch, and tolimit execution of said commands in the one or more command queues,during each said epoch, in accordance with the received average powercredit allocation and the received peak power credit allocation for theepoch.
 11. The memory system of claim 1, wherein: each respectivechannel controller of the plurality of channel controllers is configuredto determine a backlog of the respective channel controller and providethe determined backlog to the storage controller; the average powercredit received by each respective channel controller is based at leastin part on the determined backlog provided by the respective channelcontroller to the storage controller.
 12. The memory system of claim 11,wherein each respective channel controller is configured to determinethe backlog score of the respective channel controller in accordancewith a count of commands whose execution was deferred by the respectivechannel controller, in an epoch prior to a current epoch, in accordancewith a determination that executing those commands would have requiredpower credits in excess of power credits available in the respectivechannel controller during the prior epoch.
 13. The memory system ofclaim 11, wherein each respective channel controller is configured todetermine the backlog score of the respective channel controller inaccordance with pending commands in the one or more command queueswaiting for execution.
 14. A method of operation in a memory systemhaving a storage controller and a plurality of distinct sets ofnon-volatile memory devices, the method comprising: at each respectivechannel controller of a plurality of channel controllers, each channelcontroller corresponding to a distinct set of the plurality of distinctsets of non-volatile memory devices, each respective channel controllerhaving one or more command queues for holding the respective channelcontroller's pending commands: receiving power credits allocated by thestorage controller, wherein each respective channel controller isconfigured to receive an average power credit and a peak power credit;executing commands in one or more command queues, including limitingexecution of said commands in accordance with the received average powercredit and the received peak power credit, wherein each respectivechannel controller is configured to limit execution of said commandsbased on a determination that a number of received average power creditsand/or peak power credits meets excess power credit criteria; and at thestorage controller, allocating a variable total number of average powercredits and allocating a fixed total number of peak power credits. 15.The method of claim 14, including determining the total number of peakpower credits based on characteristics of the memory system.
 16. Themethod of claim 14, including adjusting the total number of averagepower credits based at least in part on one or more board powermeasurements.
 17. The method of claim 14, including adjusting the totalnumber of average power credits based at least in part on one or moretemperature measurements.
 18. The method of claim 16, further comprisingadjusting the total number of average power credits at fixed timeintervals.
 19. A memory system, comprising: a plurality of distinct setsof non-volatile memory devices; a storage controller; a plurality ofchannel controllers, each respective channel controller corresponding toa distinct set of the plurality of distinct sets of non-volatile memorydevices, each respective channel controller having one or more commandqueues for holding the respective channel controller's pending commandsand including: means for receiving power credits allocated by thestorage controller, wherein each respective channel controller isconfigured to receive an average power credit and a peak power credit;and means for executing commands in the one or more command queues,including limiting execution of said commands in accordance with thereceived average power credit and the received peak power credit,wherein each respective channel controller is configured to limitexecution of said commands based on a determination that a number ofreceived average power credits and/or peak power credits meets excesspower credit criteria.